More changes for logging and bgen write options#65
More changes for logging and bgen write options#65kianfar77 merged 9 commits intoprojectglow:masterfrom
Conversation
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com>
Codecov Report
@@ Coverage Diff @@
## master #65 +/- ##
=======================================
Coverage 94.11% 94.11%
=======================================
Files 74 74
Lines 3569 3569
Branches 331 332 +1
=======================================
Hits 3359 3359
Misses 210 210
Continue to review full report at Codecov.
|
karenfeng
left a comment
karenfeng
left a comment
There was a problem hiding this comment.
Looks good overall, just one question.
| private val ENV_PREFIX = "env_" | ||
| private val INPUT_FORMATTER_PREFIX = "in_" | ||
| private val OUTPUT_FORMATTER_PREFIX = "out_" | ||
| val LOGGING_BLOB_KEY = "pipeCmdTool" |
There was a problem hiding this comment.
Why are we moving this here?
There was a problem hiding this comment.
For all other features, like vcf/bgen/plink read/write and variant normalizer transformer, the options which are defined inside the respective packages are used as keys for the maps in the logging json blobs. I do not want this one to be an exception and be defined outside of pipe transformer. This makes the logging code cleaner on the universe side as well.
There was a problem hiding this comment.
I see. Can we move the transformer options to a higher-level object as we did for the datasource options for transparency?
There was a problem hiding this comment.
Sure I can. But if ok with you let's do it in another PR because I want to get the universe PR review moving froward with data and observability teams and that depends on this PR. Moving options to higher level can be done independently.
* Remove SpecificInternalRow buffer in RowConverter * comment Signed-off-by: Henry Davidge <hhd@databricks.com>
* minor changes needed for logging Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com> * logging plink blob and minor changes Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com> * bug fix Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com> * moving pipe blob key Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com> * drop HlsBlobKeys Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com> * changes for logging Signed-off-by: kianfar77 <kiavash.kianfar@databricks.com> Signed-off-by: Henry Davidge <hhd@databricks.com>
2 participants
What changes are proposed in this pull request?
Adds minor changes to work with logging
Put bigbegn write options in common.BgenOptions